Automated grossing image synchronization and related viewers and workstations

ABSTRACT

Methods, systems, workstations, viewers, grossing stations and computer include a display and a viewer circuit configured to cause the display to concurrently display: (i) a macroscopic view of a grossing specimen with virtual cut location marks associated with orientation and location of actual physical cut locations used to obtain tissue samples from the grossing specimen and (ii) at least one digital microscopic whole-slide image (WSI) of a tissue section from the specimen. The display can show the at least one WSI image on the display with a relevant cut location mark on the macroscopic view shown visually enhanced from other cut location marks on the macroscopic view to thereby allow a user to visually connect where the tissue section in the WSI image was obtained during a grossing procedure.

RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalApplication Ser. No. 61/970,608, filed Mar. 26, 2014, the contents ofwhich are hereby incorporated by reference as if recited in full herein.

FIELD OF THE INVENTION

The present invention is related to medical microscopy imaging withinthe digital pathology domain.

BACKGROUND

In the histopathology workflow, tissue samples can be cut or otherwiseexcised from a gross specimen, a procedure known as “grossing”. These(sub)samples then go through a number of processing steps in thelaboratory, typically ending up as stained tissue sections mounted onglass slides. In a digital pathology workflow, these glass slides arethen scanned into digital histology microscopy images. While thepathologist (a specialist physician) performs the microscopic imagereview, the grossing is frequently done by other staff, such astechnicians, resident physicians or other pathologists. For an overviewof grossing procedures, see Theory and Practice of HistologicalTechniques, John D. Bancroft, Elsevier Health Sciences, the content ofwhich are hereby included by reference as if recited in full herein.

In histopathology, the connection between a respective gross (or“grossing”) specimen and corresponding histology slides can be of vitalimportance. A macroscopic examination is done for most specimens anddiagnostic findings are documented. In order to relate to these findingsduring microscopic review, the location of the findings relative to arespective tissue specimen are often marked with colored dye during thegrossing procedure. The dye is preserved in the tissue samples throughthe subsequent processing and shows up in the microscopic images of thetissue sections on the glass slides. The location of the cut in thespecimen is also of high importance. Knowing both the macroscopicorientation and the location of the tissue section relative to thetissue specimen are particularly important when assessing margins. Forexample, if cancerous cells are found in a region close to a resectionborder, the risk is high that there are remaining cancerous cells in thepatient. Thus, the relation between macroscopic and microscopic findingsis important and often affects therapy and medical decisions such astherapeutic treatment selections for the patient.

To meet the informational need, the pathologist performing themicroscopic review should be provided a description of the grossspecimen. In the past, such information was in the form of a manuallydrawn sketch on paper or in the form of a digital photograph(s) of thespecimen from a macroscopic camera. If there is such a macroscopiccamera, it may operate with software that allows manual drawing of cutsand other annotations such as measurements on the gross image.

Unfortunately, manually entered physical or digital markings on themacroscopic images can be relatively cumbersome to make, especially inthe wet and dirty environment of a grossing station, and they can beimprecise or subjective due to the manual nature of the marking.Furthermore, at the microscopic review, the pathologist is required tocognitively make the connections between the markings and the scannedslides and regions of the slides.

SUMMARY

Embodiments of the invention provide improved grossing systems, viewersand workstations that can provide more intuitive viewer options and/oralternative image processing that can facilitate efficient histologyreviews.

Embodiments of the invention provide automated electronic marking ongross images and an image processing method, circuit, viewer, grossingstation and system providing image synchronization of slides of tissuesections automatically correlated to a specimen and a physical locationon a macroscopic image(s) for histopathology.

A histopathology and/or cytopathology viewer, comprising: a display; anda viewer circuit configured to cause the display to concurrentlydisplay: (i) a macroscopic view of a grossing specimen with virtual cutlocation marks associated with orientation and location of actualphysical cut locations used to obtain tissue samples from the grossingspecimen and (ii) at least one digital microscopic whole-slide image(WSI) of a tissue section from the specimen. For the at least one WSIimage on the display, a relevant cut location mark is shown visuallyenhanced from other cut location marks on the macroscopic view tothereby allow a user to visually connect where the tissue section in theWSI image was obtained during a grossing procedure.

The viewer circuit can be configured to visually change a displayparameter of at least one of: (a) a virtual cut location mark on themacroscopic view that is associated with a location on the specimen fromwhich a tissue section of the at least one WSI in the viewer display wasobtained; or (b) virtual cut location marks on the macroscopic view thatare not associated with the tissue section of the at least one WSI inthe viewer to thereby visually enhance the relevant cut mark location.

The viewer may be configured to provide thumbnail WSI images of slidesassociated with the grossing specimen of a patient and visually change adisplay parameter of a thumbnail associated with the at least one WSI inthe viewer.

A perimeter of the thumbnail or thumbnails associated with the at leastone WSI in the viewer display can be visually enhanced and/or otherthumbnails are dimmed or faded in the display window. The virtual cutlocation marks can be provided as overlay linear objects on themacroscopic view. A virtual cut location mark on the macroscopic viewthat is associated with a location on the specimen from which a tissuesection of the at least one WSI in the viewer display was obtained canbe visually highlighted, increased in intensity or changed in colorrelative to other virtual cut mark locations to thereby visually connectthe WSI image in the view to a correct virtual cut location mark.

The viewer circuit can be configured to analyze scanned WSI images fordetermining color and color location and adjust an orientation of theWSI image for the viewer to consistently provide views of the WSI imagesto the display in a common orientation with respect to a tissue samplelocation and orientation in or on the grossing specimen based on thedetermined color.

Other embodiments are directed to a grossing workstation. Theworkstation includes: a cut mark location identification circuit; atleast one camera over a workspace at the workstation in communicationwith the cut mark location identification circuit; and a cuttinginstrument in communication with the cut mark location identificationcircuit. The cut mark location identification circuit can be configuredto identify physical cut mark locations on a grossing specimen andgenerate corresponding virtual cut location marks on a macroscopic imageor model of the specimen for a viewer. A respective virtual cut locationmark can correspond to a physical cut location made by the cuttinginstrument on the grossing specimen used to obtain a tissue sample suchthat the virtual cut location marks are given a correct spatial positionon the macroscopic image or model of the specimen for the viewer.

The cut mark location circuit can analyze a plurality of macroscopicimages of the specimen obtained by the at least one camera during agrossing procedure of the specimen, including at least one basemacroscopic image obtained prior to a physical cutting to obtain atissue sample. The cut mark location identification circuit can beconfigured to electronically place the virtual cut location marks asoverlay cut location objects on the macroscopic image or model of thespecimen for the viewer.

The cut mark location identification circuit can be configured toelectronically track movement of the cutting instrument in theworkspace.

The cut mark location identification circuit can be configured to employimage recognition of movement and location of the cutting instrument inimages obtained by the at least one camera to generate the virtual cutmark locations on the macroscopic image or model.

The cutting instrument can be configured with a defined leading endportion shape which has a distinct conspicuous visual appearancedetectable by image recognition to allow the cut mark locationidentification circuit to identify movement and contact of the cuttinginstrument for identifying the placement and orientation of therespective virtual cut location marks.

The cut mark location circuit can be configured to employ a hands-freecommand or input from a user to initiate identifying a cut location.

Other embodiments are directed to methods of obtaining and/or processingdigital pathology and cytology images for viewing. The methods include:electronically identifying when a cutting instrument is approachingand/or on a grossing specimen; and automatically electronicallygenerating virtual cut location marks on a digital macroscopic image ormodel of a grossing specimen based on the electronic identification,wherein a respective virtual cut location mark corresponds to a physicalcut location made by the cutting instrument on the grossing specimenused to obtain a tissue sample such that the virtual cut location isgiven a correct spatial position on the macroscopic image or model ofthe specimen.

The method can include electronically automatically obtaining aplurality of macroscopic images of the specimen during a grossingprocedure of the specimen, including at least one base macroscopic imageobtained prior to one or more physical cutting to obtain a tissuesample, and placing the virtual cut location marks on one or more of theobtained macroscopic images or model of the specimen for display in aviewer.

The electronic identification can be carried out by electronicallytracking movement of the cutting instrument.

The electronic identification can be carried out using image recognitionof movement and location of the cutting instrument in the obtainedimages.

The electronic identification of the cutting instrument can be carriedout using image recognition of a defined leading end portionconfiguration which is configured as to have a defined distinctconspicuous visual appearance detectable by image recognition.

The electronic identification can be initiated with a hands-free commandto a cut location identification circuit in communication with agrossing workstation.

The electronic identification can be initiated by the cutting instrumenttransmitting a wireless signal at a time of a cut.

The automatic electronic generation of the virtual cut location markscan include electronically generating an electronic overlay of at leastone cut location object provided as a linear mark and displaying theoverlay on a macroscopic digital image or model of the specimen.

The method can also include concurrently displaying (i) a macroscopicimage of the specimen with the virtual cut location marks with (ii) atleast one digital microscopic whole-slide image (WSI) of tissue from thespecimen. A virtual cut location mark associated with the at least onedigital WSI can be visually enhanced relative to other cut locationmarks when it is associated with where a tissue section from the WSI wasobtained from the specimen.

The virtual cut location marks can be shown as respective overlays onthe macroscopic image with the visual enhancement provided byhighlighting a virtual cut location mark whenever it corresponds to amicroscopic WSI image in a viewer.

The method may include electronically automatically obtaining aplurality of macroscopic images of the specimen during a grossingprocedure of the specimen, including at least one base macroscopic imageobtained prior to any physical cutting to obtain tissue samples, andplacing the virtual cut location marks on one or more of the obtainedmacroscopic images or a model of the specimen for a viewer macroscopicimage; and may also include electronically adjusting for movement of thespecimen during a grossing procedure using the base image and one ormore of the subsequent plurality of images to register respectivevirtual cut locations to the viewer macroscopic image or model.

It is noted that any one or more aspects or features described withrespect to one embodiment may be incorporated in a different embodimentalthough not specifically described relative thereto. That is, allembodiments and/or features of any embodiment can be combined in any wayand/or combination. Applicant reserves the right to change anyoriginally filed claim or file any new claim accordingly, including theright to be able to amend any originally filed claim to depend fromand/or incorporate any feature of any other claim although notoriginally claimed in that manner. These and other objects and/oraspects of the present invention are explained in detail in thespecification set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is schematic illustration of a grossing station accommodatingautomated visual representations of cut mark locations on a macroscopicimage corresponding to cut locations of actual tissue samples from aspecimen according to embodiments of the present invention.

FIGS. 2A and 2B are schematic illustrations of additional exemplarygrossing workstations for automatically, electronically obtaining cutmark and/or dye locations during a grossing procedure according toembodiments of the present invention.

FIG. 3 is a schematic illustration of a grossing system that candigitally correlate cut mark location with slides of tissue samples fora viewer according to embodiments of the present invention.

FIG. 4 is a screen shot of a display of an exemplary viewer providing amicroscopic image concurrently with a macroscopic image with one or morecut mark locations shown and/or with a current cut location associatedwith a tissue section of a WSI in a large window or portion of a displayaccording to embodiments of the present invention.

FIG. 5 is a schematic illustration of a grossing workstation that canprovide electronic data of dye mark locations on a grossing specimenimage for display on a macroscopic image or to be able to correlateviewer user views of WSI microscopic slides according to embodiments ofthe present invention.

FIG. 6 is a partial schematic illustration of a combined cuttinginstrument and dye marking applicator according to embodiments of thepresent invention.

FIG. 7 is a schematic illustration of yet another embodiment of agrossing station for identifying cut mark and/or dye locations on aspecimen and generating virtual marks on a macroscopic image and/orcorrelating microscopic views to facilitate user comprehension ofmicroscopic tissue sections to the grossing specimen according toembodiments of the present invention.

FIGS. 8-10 are screen shots of examples of viewer display windows withcorrelated microscopy images for histology analysis according toembodiments of the present invention.

FIG. 11 is a flow chart of an exemplary workflow process to digitallyprovide cut mark location data for correlating microscopic slide viewsto physical cut locations according to embodiments of the presentinvention.

FIG. 12 is a flow chart of an exemplary workflow process to digitallyprovide dye location on a specimen to digitally correlate microscopicslide views according to embodiments of the present invention.

FIG. 13 is a schematic illustration of a viewer with user interfaceselections optionally in communication with a server and optionalmedical and/or laboratory record database according to embodiments ofthe present invention.

FIG. 14 is a schematic illustration of a circuit comprising a serveraccording to embodiments of the present invention.

FIG. 15 is a schematic illustration of an exemplary data processingsystem according to embodiments of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, in which some embodiments of theinvention are shown. This invention may, however, be embodied in manydifferent forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art. Likenumbers refer to like elements throughout. It will be appreciated thatalthough discussed with respect to a certain embodiment, features oroperation of one embodiment can apply to others.

In the drawings, the thickness of lines, layers, features, componentsand/or regions may be exaggerated for clarity and broken lines (such asthose shown in circuit or flow diagrams) illustrate optional features oroperations, unless specified otherwise. In addition, the sequence ofoperations (or steps) is not limited to the order presented in theclaims unless specifically indicated otherwise.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Like numbersrefer to like elements throughout. In the figures, the thickness ofcertain lines, layers, components, elements or features may beexaggerated for clarity. As used herein, phrases such as “between X andY” and “between about X and Y” should be interpreted to include X and Y.As used herein, phrases such as “between about X and Y” mean “betweenabout X and about Y.” As used herein, phrases such as “from about X toY” mean “from about X to about Y.”

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the specification andrelevant art and should not be interpreted in an idealized or overlyformal sense unless expressly so defined herein. Well-known functions orconstructions may not be described in detail for brevity and/or clarity.

It will be understood that when a feature, such as a layer, region orsubstrate, is referred to as being “on” another feature or element, itcan be directly on the other element or intervening elements may also bepresent. In contrast, when an element is referred to as being “directlyon” another feature or element, there are no intervening elementspresent. It will also be understood that, when a feature or element isreferred to as being “connected” or “coupled” to another feature orelement, it can be directly connected to the other element orintervening elements may be present. In contrast, when a feature orelement is referred to as being “directly connected” or “directlycoupled” to another element, there are no intervening elements present.The phrase “in communication with” refers to direct and indirectcommunication. Although described or shown with respect to oneembodiment, the features so described or shown can apply to otherembodiments.

The term “circuit” refers to software embodiments or embodimentscombining software and hardware aspects, features and/or components,including, for example, at least one processor and software associatedtherewith embedded therein and/or executable by and/or one or moreApplication Specific Integrated Circuits (ASICs), for programmaticallydirecting and/or performing certain described actions, operations ormethod steps. The circuit can reside in one location or multiplelocations, it may be integrated into one component or may bedistributed, e.g., it may reside entirely in a workstation or singlecomputer, partially in one workstation, cabinet, or computer, or totallyin a remote location away from a local display at a workstation. If thelatter, a local computer and/or processor can communicate over a LAN,WAN and/or internet to transmit patient macroscopic and microscopicimages.

The term “automatically” means that the operation can be substantially,and typically, entirely, carried out without human or manual input, andis typically programmatically directed and/or carried out. The term“electronically” includes both wireless and wired connections betweencomponents. The terms “display” and “screen” are used interchangeably.

The term “programmatically” means that the operation or step can bedirected and/or carried out by a digital signal processor and/orcomputer program code. Similarly, the term “electronically” means thatthe step or operation can be carried out in an automated manner usingelectronic components rather than manually or using merely mental steps.

The term “clinician” refers to a pathologist, physician, oncologist, orother personnel desiring to review tissue sample data of a subject,which is typically a live human or animal patient but forensic uses arealso contemplated.

The term “user” refers to a person or device associated with thatperson, that uses the described and/or claimed feature or item, such asa technician, pathologist or other expert, or clinician or even apatient.

The term “about” means that the recited parameter can vary from thenoted value, typically by +/−20%.

The term “PACS” refers to a PICTURE ARCHIVING AND COMMUNICATION SYSTEM.

The term “magnification” means the image resolution measured inmicrometers per pixel, applicable both for the scanned image and theimages displayed on screen. Higher magnification corresponds to a lowermicrometer per pixel value than lower magnification and vice versa.

The term “high magnification” means displaying an image with anon-screen magnification relatively close to the magnification with whichthe original image was scanned. Current (2013) clinical scanningprotocols commonly employ 200 times or 400 times magnification,corresponding to 0.5 and 0.25 micrometers per pixel respectively. Inthis case “high magnification” corresponds to magnification range ofbetween about 0.25-1.0 micrometers per pixel.

The term “low magnification” means displaying an image with an on-screenmagnification substantially lower than the magnification with which theoriginal image was scanned. In the case of using a scanning protocol of0.5 or 0.25 micrometers per pixel, “low magnification” corresponds tomagnification range of about 2 micrometers per pixel and above, forexample 10 micrometers per pixel.

The term “visually enhanced” means that at least one display parameteris adjusted so that one or more noted features are visually dominantrelative to other related features, e.g., cut mark locations and/orthumbnail images, for example. The display parameter can comprisebrightness, opacity, color, color saturation and the like. Either thedisplay parameter of the visually enhanced feature is adjusted and/orthe display parameter of a non-relevant feature is adjusted, e.g., thevisually enhanced feature can be shown with increased brightness,intensity or a bolder or prominent color and/or the non-relevantfeatures can be shown with reduced brightness and/or intensity or in aless prominent color.

The term “model” refers to a rendered representation of the grossingspecimen rather than an actual image obtained by a camera.

A grossing workstation refers to a workstation where gross examinationor “grossing” is carried out by which pathology “grossing specimens” areinspected, typically with the naked eye, to obtain diagnosticinformation, while being processed for further microscopic evaluation.There are usually two end products of the gross examination of asurgical specimen. The first is the gross description, a document whichserves as the written record of the examiner's findings, and is includedin the final pathology report. The second product is a set of tissueblocks, typically postage stamp-sized portions of tissue sealed inplastic cassettes, which will be processed into slides for microscopicexamination.

According to the instant application, the grossing workstation can beconfigured to obtain or capture at least one macroscopic image of agrossing by at least one digital camera. The term “macroscopic image”refers to an image of a the grossing specimen that is to be evaluatedand from which discrete tissue samples will be obtained for furtherprocessing and digital microscopic WSI of thin tissue sections. The WSIcan be described as virtual microscopy for cytopathology.

A “grossing specimen” (“specimen”) is a piece of tissue/organ forpathology evaluation, which can be as large as an entire breast or largeportion of an intestine. Small pieces of tissue that are obtained (e.g.,cut) at grossing can be called “samples” of a specimen. It is also notedthat some specimens are so small when they arrive at a pathologyevaluation site (such as prostate screening biopsies, which are just athin thread) that no grossing is necessary. These small specimenstypically just proceed to the next step in the laboratory processing.The large specimens (and respective tissue samples) and small specimenscan all be referred to as “biopsies.” Tissue samples from a specimen ora small specimen itself can be put in one or more paraffin blocks as iswell known to those of skill in the art. From a block, micrometer-thinsections (slices) are generated and put onto glass slides for WSIevaluation or virtual microscopy. The present application isparticularly useful for specimens requiring grossing “G.” Thus, forpurposes of the instant application, a “tissue sample” refers to piecesobtained (e.g., cut) from a specimen G during grossing. The slidescomprise thin sections of one or more tissue samples from one or moreblocks.

During processing both dyes and stains can be used. A dye is applied atgrossing to identify an excision or cut site for a particular tissuesample, or, more commonly, the orientation of the sample with respect tothe specimen and other anatomy. Staining is applied to the sections.Typically, the glass slides are immersed or otherwise exposed to adefined stain to make certain characteristics visually appear in atissue section for a microscopic review.

The WSI of thin tissue sections can be digital images of a glass (orother suitable substrate) slide with a thin section of tissue depictedby a medical microscope. The WSI can be high resolution and have betweenabout 1×10³-1×10¹² pixels, and a magnification typically about 0.1-1.0micrometers per pixel. Today there are many scanners capable ofproducing high-quality digital images from microscopy glass slides. See,e.g., Rojo et al., Critical comparison of 31 commercially availableslide systems in pathology, Int J Surg. Pathol., 2006; 14(4):285-305,the contents of which are hereby incorporated by reference herein. Theresulting WSI digital images can be very large, for instance30,000×40,000 pixels, 100,000×100,000 pixels or more. In histology, atwo-dimensional (2D) image often suffices, but there is also thepossibility to produce slices across the depth of the tissue section,creating a three-dimensional (3D) dataset even though the extent in thez-direction can be far different from the x-y directions.

The term “viewer” refers to an electronic interface that allows a userto select to display different images and different magnification levelsof target tissue, typically tissue associated with WSI.

The term “virtual cut location marks” refers to electronically generatedrepresentations of physical cut mark locations placed on a macroscopicimage or model of a grossing specimen to represent actual physical cutlocations made on a specimen to acquire tissue samples. The term“virtual color location marks” refers to electronically generatedrepresentations of physical color locations placed on a macroscopicimage or model of a grossing specimen to represent actual physicallocations where color is applied to the grossing specimen associatedwith acquired tissue samples. The virtual cut location and/or colormarks can be provided as electronic objects onto an actual macroscopicimage of the specimen or a macroscopic model of the specimen.

The term “macroscopic” refers to a view of the grossing specimen itself(image or model) rather than a microscopic view of thin tissue slices.

Embodiments of the invention recognize that image analysis tools inpathology/cytology can be challenging as they should fit well into theroutine clinical workflow. In comparison to a research lab setting, thismeans additional requirements of precision, robustness and performance(e.g., throughput speed). For instance, waiting a few minutes for animage processing algorithm to finish is unfeasible, whereas that may beconsidered quite reasonable in a research setting.

Referring to FIG. 1, an exemplary grossing system 100 with a grossingworkstation (or “station”) 105 is shown. The grossing station 105includes, for example, a cutting region 106, typically provided with acutting board or plate 12 and at least one camera C for obtaining one ormore images of a grossing specimen G of patients. The at least onecamera C is typically a macroscopic camera such as PathStand™ 24available from Diagnostic Instruments, Inc, Sterling Heights, Mich.,USA.

The grossing system 100 can also include a cut mark circuit 30 that isconfigured to automatically, electronically identify physical cutlocations to define a respective physical cut location 10 (10 i to 10 n)associated with a current location of an excision tool 20, such as ascalpel or knife, at time (t) in relation to the grossing specimen G.The circuit 30 can render a visual virtual representation of arespective physical cut location 110 ₁ (e.g., a virtual cut locationmark) at a correct spatial location on one or more macroscopic images Mof the specimen G which can be shown on a display 35 (during the cuttingand/or at a viewer V (FIG. 4), at or downstream of the grossing station)105.

The circuit 30 can be configured to provide the at least one macroscopicimage M with the virtual cut mark representations 110 ₁ to 110 n to oneor more displays 35. The number “n” typically means between 2-50 andcorresponds to the number of physical excisions and associated cut markrepresentations made on a respective specimen G.

As shown in FIG. 3, the work flow is carried out carefully to track thespecimen G and resultant tissue samples Ts and sections S in thelaboratory: The typical situation is that there are barcodes 40generated/provided or updated at each step. When a tissue sample Ts iscut, it goes into a bar-coded cassette C, which is correlated orregistered to being “block X of specimen Y” from cut location “1.” Whentissue sample section(s) from a block B goes onto a glass slide S, thebarcode of the slide S can be correlated and/or registered to being“glass Z of block X.” The glass slide barcode 40 can be scanned by thescanner with a respective tissue section on the slide S for a digitalmicroscopy WSI or digitally read by a digital reader or even manuallyinput to correlate and/or register the identifier data during digitalscanning via a scanner of respective slides S. The digital WSI image canbe identified, correlated and/or registered as “WSI W=glass Z of blockX.” The specimen G, block B, and slide S to cut location datacorrelations can be held in a data record file 104 of a database managedby an electronic processing system such as, but not limited to, a typeof information system called LIS or LIMS (Laboratory Information[Management] System) as is well known to those of skill in the art. TheWSI from the scanner can include dye colors on respective slides S whichcan be digitally identified (green on right side or end of slide, forexample).

The bar code 40 can comprise one or more of a 1-D, 2-D (UPC or QuickResponse code or “QR”, for example) or even a 3-D bar code. Differenttypes of bar codes can be used for different steps in the work flow. Itis also contemplated that the system 100 can be configured to employother devices/ways of providing the correlation and/or tracking data,such as, for example, RFID tags, magnetic strips or smartcards (such asused with credit cards) with a microprocessor, for example. Associatedreaders and processing systems can be integrated into the LIS system orgrossing synch circuit 100.

The circuit 30 can be configured to correlate virtual cut mark locations110 on the macroscopic map M (FIG. 4) to the tissue samples Ts takenfrom the respective cut locations 10 on the specimen G. Thus, forexample, different tissue samples from different cut locations can haveunique electronic identifiers that correlate a tissue sample Ts with aphysical cut location 10 and/or the corresponding virtual cut location110. This correlation data can be subsequently integrated with blockidentifier data (paraffin block of the tissue sample). Thus, the circuit30 can be configured to provide information to a grossing synch system100 that provides cut location data for a respective slide, includingits virtual physical cut location 110 on a macroscopic image M.

In some embodiments, the electronic database record 104 a slide S of atissue section can include a data stream that identifies a patient, aspecimen G, and tissue sample cut locations 10 and/or 110, and a slidesection alphanumeric identifier associated with the S (WSI) on a viewerV.

FIG. 4 illustrates an exemplary viewer V with a display 35 that shows amacroscopic image M with the virtual cut marks 110. A WSI view of aslide S is shown with the macroscopic image M in a smaller windowadjacent or overlaid on the WSI view of the slide S. The system 100 canbe configured to visually indicate which cut location mark 110 a isassociated with the current WSI slide, e.g., current slide Sa isassociated with visually enhanced cut location mark “A.” The cutlocation marks 110 may be represented as linear overlay objects and/orlines with a unique alphanumerical label as shown. The relevant cutlocation mark 110 a associated with the currently viewed slide Sa can bevisually enhanced relative to other cut location marks 110, e.g., shownin bold, with increased intensity and/or opacity and/or in a differentcolor from non-relevant cut marks. The viewer V may be configured toshow only the relevant cut mark 110 a and omit or greatly reduce theintensity and/or visual prominence of the non-relevant cut marks (thecut marks not associated with the current slide Sa on the display 35).

The viewer circuit Vc may be configured to visually enhance a relevantphysical cut location (e.g., highlight and/or apply a color or border tothe cut location) in a macroscopic image M that corresponds to the slidein the viewer V rather than apply overlays or location identificationobjects.

The cut location marks 110 on a respective macroscopic image or model Mmay be configured as active or inactive objects or links. For activeobjects or links, a user may select (e.g., click or touch) a particularcut location mark 110 on image or model M on the display 35 and theviewer V can automatically present WSI slides S associated with theselected cut location mark. The resulting presentation on the display 35can be to provide the relevant slides S in a new larger viewing window,in a concurrent adjacent or overlay window and/or along one side of themacroscopic image M. A set of thumbnail WSI images 200 of differentslides S may be concurrently displayed along a side, top or bottom ofthe screen. The slide Sa being currently viewed by the viewer V can behighlighted, shown as color coded perimeter matching a color of thevisually enhanced cut 110 a (cut A). The set of slides 200 may bedisplayed via a pull down option, for example, or in subsets of slides,relative to a particular block B or a particular cut location, forexample.

A UI 130 (FIG. 13) associated with the viewer V can be configured toallow a user to customize viewing sets or select predefined groups forfacilitating faster reviews. For example, all slides from the same cutlocation, typically some with different stains, may be grouped together,e.g., adjacent each other, shown only in the thumbnail set 200 (withother slides from other cut locations omitted) and/or concurrently shownwith all slides but emphasized with a common color background and/orperimeter, with an adjacent icon or with an overlay. In someembodiments, each slide S associated with a respective cut location mark110 can be color-coded to the mark of that location, e.g., the objectoverlay line can have a color that is the same as that of the slidebackground and/or perimeter border, for example. Thus, each mark 110 canbe shown with a defined color that corresponds to the slide colorindicia, e.g., each mark 110 and its associated slides S can be shownall in green, red, fuchsia, pink, yellow and the like.

Still referring to FIG. 4, the viewer V may optionally also beconfigured to provide a reference window Wref which shows an overview ofthe entire currently viewed slide Sa.

There may be more than one macroscopic image M for a specimen G. Forexample, the specimen G may be turned over for additional tissue samplesfrom different cut locations. The circuit 30 can generate acorresponding number of macroscopic images M to show the associated cutmark locations 110. Also, it is contemplated that, when cutting thespecimen G, its shape may change. Thus, in some embodiments, the circuit30 can be configured to employ image registration to adjust for shapechanges and other accidental movement. Thus, for example, a first orbase image can be obtained. Then, whenever a cut (and/or dye mark orinking as will be discussed below) is made, a new macroscopic image canbe obtained/captured. An image registration module (e.g., software) canidentify spatial transformation (warping) of the second or subsequentimage that is necessary to fit the base image. That spatialtransformation can then be applied to the location of the new virtualcut mark 110, such that it becomes correctly related to the base imageM. There are many alternative methods that could be suitable for themacroscopic image registration, for example Bowen, F.; Du, E.; JianghaiHu, “New region feature descriptor-based image registration method,”Systems, Man, and Cybernetics (SMC), 2012 IEEE International Conference,vol., no., pp. 2489, 2494, 14-17 Oct. 2012, the contents of which arehereby incorporated by reference as if recited in full herein.

Different specimens G from the same or different patients may havedifferent numbers of cuts 10 i-10 n at different locations betweenspecimens G and/or specimen types; thus, the corresponding virtual cutmarks 110 ₁-110 n can reside at different locations in the macroscopicimage M of the specimen G.

At the grossing station 105, when a user wants to make a cut 10 andcreate an associated virtual or digital cut mark 110 location with acorrect physical location for the image M, he/she can initiate a cuttingand, concurrently, the circuit 30 can, automatically or based on inputfrom a user interface (UI) associated with the circuit 30, initiate oractivate the circuit 30, to generate data to and/or make a correspondingmarking 110 for a macroscopic image M, while the user is holding thecutting instrument (e.g., scalpel) 20 at cut location 10 on the specimenG. Thus, the circuit 30 can be in communication with at least one cameraC at the workstation 105 to obtain at least one macroscopic imageconcurrent with a physical cut of the specimen for a respective tissuesample.

The circuit 30 can generate a virtual mark 110 of a respective cut inreal time or near real time, e.g., while a corresponding physicalcutting is carried out, typically within 0.001 second to about 1 secondof actual initiation of a respective cutting or upon contact of acutting instrument 20 to the specimen G. However, in embodiments, thecircuit 30 can generate the virtual markings 110 later, e.g., after thespecimen G leaves the grossing station 105, based on data collectedduring the grossing procedure of the specimen G.

In some embodiments, no separate marking tool is required for thevirtual marking. For example, a cutting instrument 20 (e.g., scalpel)used for the specimen cutting can also be configured to generate thevirtual mark 110. In some embodiments, the cutting instrument 20 caninclude a circuit interface 20 i. For example, the cutting instrument 20can comprise a user interface UI that communicates with the circuit 30.The UI may be configured as a hand, finger or thumb input. The UI maycomprise a pressure sensor. Thus, for example, as a clinician presses onthe cutting implement 20 to cut tissue, an increase in pressure or forceon the input 20 i can generate a signal to the circuit 30 that actualcutting is occurring, thus triggering a camera shot and/or marking of avirtual cut location 110 corresponding to the physical cut location 10.

Referring to FIG. 2A, in some embodiments, the cutting instrument 20 canhave various circuit interface configurations for communicating when andwhere to make a virtual mark 110. For example, the cutting instrument 20can comprise a sensor 20 s on a bottom tissue contacting end portionthat contacts tissue to complete a circuit, close a switch or detectforce or pressure upon contact/cutting action. In some embodiments, theinterface 20 i can comprise a UI 20 ui which can be positioned as afinger or thumb position that allows an easy-to-use interface that canbe carried out in response to when a user initiates a cutting force witha finger or thumb. In some embodiments, the cutting instrument 20 caninclude at least one fiducial or optical signal feature 20 o that can beinterrogated in a workstation 105 (in a defined volumetric space) and/orin image data to identify when a cut is occurring and/or a cut marklocation. In some embodiments, to initiate the virtual marking 110, thecircuit 30 can be configured to respond to a hands-free command signal20 hf, for instance by pressing a foot pedal 20 f or using avoice-activated command (audio input) 20 a. Combinations of thedifferent circuit interfaces 20 i may be used.

In some embodiments, a separate marking tool may be used to mark the cutlocation on the specimen after tissue sample is excised. Thus, the toolmay be configured as a pen-like device that traces over a cut on thespecimen G to generate the virtual cut marking location 110.

FIG. 2A also illustrates that a grossing workstation 105 can compriseoptical detectors D that may be used with one or more cameras C todetermine when a cutting instrument 20 is in or approaching the specimenvolume G to obtain an image and/or generate the cut mark location 110.The detectors D may be optical, e.g., infrared sensors, for example,with transmitters/emitters and receivers positioned across the specimenG.

FIG. 2B illustrates a grossing workstation 105 with a grid 120 forminggraduated, incremental markings for a volumetric, Cartesian, coordinatesystem. The grid 120 can be used to define coordinates of respective cutmarks in the workspace at the workstation 105 that can be electronicallyconverted to the virtual markings 110. The cutting board or plate 106may optionally be used to provide X and Y axis markings in the plane ofthe board or plate 106. A macroscopic image M can be generated with avirtual box-like grid 120 v in the virtual space to identify the cutmarks 110 on the display 35. The grid 120 may be faded or turned on andoff on the display via the system 100 such as the location markingcircuit 30 or another module via a Ill.

The tracking of the cutting instrument 20 and/or identification ororientation and length of a particular cut mark can be carried out inseveral ways. One possibility is to use image analysis techniques. Forexample, a first pre-cut (base) image of the specimen G can be captured.Referring again to FIG. 1, the top of the cutting tool 20 u can comprisea visually recognizable or enhanced (e.g., conspicuous) color, LED orother optical feature and/or a defined shape that can be identified byimage recognition, for example. When a cut 10 is made on the specimen G,another image can be captured. An image analysis circuit/module (e.g.,software) can be used to identify the location of the scalpel top 20 uand create a digital virtual cut location mark such as a “cut referenceobject” referring to a location in the base image. The “cut referenceobject” can be shown as a marker on the base or pre-cut image (which canbe registered to a subsequent macroscopic image M or can be themacroscopic image or model M for the viewer).

Other tracking possibilities involve other ways of locating a cuttinginstrument (e.g., scalpel blade), such as using infrared or magnetictracking devices, e.g., tracking coils opto-electronic transmitters,transponders, GPS tracking and the like.

The circuit 30 can be configured to identify a location in 3-D spacewith a coordinate system or with relative measurements of location togenerate the virtual cut mark representations 110 with a properorientation, length, width and position on the macroscopic image Mcorresponding to a respective physical cut on the specimen G.

Referring to FIG. 5, in some embodiments, the grossing sync system 100can alternatively or additionally be configured to convert a region ofthe specimen G marked with color into a digital object with reference toa location of the macroscopic image M. When color is applied via anapplicator 25 to an area or volume 15 of the specimen G (a.k.a. inking),a user and/or a dye mark circuit 31 initiates creation of a digital“dye” or “color” reference object 115 (which can also be described as a“virtual color location maker”).

The grossing system 100 may be configured to correlate a slide S to anapplied color mark. The colors (e.g., dyes) are typically not involvedin keeping track of blocks B (FIG. 3) or tissue samples on slides S(FIG. 3). However, the color (e.g., dye) is typically visible both at amacroscopic level and in the sections viewed in the microscope. Thus,where the color is associated with a dye, the dye is visually apparentin a WSI image viewed on a display or screen. The color helps areviewer, e.g., a pathologist, when looking in the microscope/WSI viewerto relate the orientation to the macroscopic situation. The pathologistor other reviewer looks to the section, the dye color and a macroscopicimage to assess how the WSI view of a section S on display relates tothe specimen, e.g., “Is this side facing towards organ X?” To relate theslide to the orientation, the color dye can provide the information,e.g., “ . . . it has blue dye and now that I look at the counterpartmacroscopic image I can see blue dye at this location which identifieswhich side of the specimen G it corresponds to.”

The macroscopic location associated with an applied color can beconveyed through a macroscopic image or photograph of the specimen.Conventionally, the location is perhaps more commonly just denoted astext in a macroscopic sketch of the specimen drawn by hand at grossing.The dye color/side correlation may be based on standardized dye markingroutines, e.g., one side of this type of specimen is always dyed green.Thus, while the color is typically applied via dyes, inks or other colorsubstances may be used whether in liquid, powder, gel, spray or otherformulation.

The system 100 and/or circuit 31 can be configured in several differentmanners to carry out the dye color representation on the macroscopicimage M by electronically identifying the location of the inked area(s)15. As for the cutting instrument 20 discussed above, the color (e.g.,dye) applicator 25 itself can be used along the lines of any of thetechniques and features discussed above. For example, the dye area canbe identified through a defined visually enhanced (contrasting orconspicuous) color and/or a defined shape of the dye applicator tip, bymagnetic or infrared tracking at the workstation 105. The dyeidentification can be configured to electronically, automaticallydistinguish between different colors. As for the above, instead of orwith tracking configurations, image analysis and/or image recognitionprotocols can be used. Thus, for example, an image of the specimen G canbe obtained concurrently with the dye application and through imageanalysis the system can identify an inked area and which color that hasbeen used.

The initiation of the virtual dye marking 120V can be performedsimilarly to the cut location. In some embodiments, the initiation canbe done automatically without manual initiation. For example, the system100 or circuit 31 can be configured to continuously capture images overa grossing procedure and/or over a portion of a grossing procedure (suchas when a dye applicator approaches the specimen similar to the cuttinginstrument discussed in FIGS. 2A and 2B, for example). When a color(e.g., dye) applicator 25 is identified while a new color appears on thespecimen G, a virtual color location mark 115 can be made. In someembodiments, the applicator 25 can be configured to change appearance ina visually detectable manner when color (e.g., dye) is released, whichcan be identified through image analysis. In some embodiments, theapplicator 25 can transmit a signal when dispensing dye.

FIG. 6 illustrates that a hybrid cutting instrument 20′ with a cuttingsurface 20 which includes an onboard color applicator 25′ with adispenser 25 d positioned proximate the cutting surface 20 c. Thus, theinstrument 20′ can be used to both cut tissue samples 10 and dispensecolor 25 d, either serially or concurrently. The applicator 25′ can beconfigured as a rotatable cartridge of different colors that can beautomatically incremented with different cuts or may be user selectedwith a user input similar to a color-selectable ink pen or marker. Thehybrid instrument 20′ can be configured with a circuit interface 20 i togenerate the cut marks 110 and the color marks 115 on a macroscopicimage or model M.

FIG. 7 illustrates an exemplary grossing workstation 105 configured witha different workflow protocol for identifying cut locations 110 and/ordye marks 115 (although discussed with respect to the cut locations). Asshown, a planning image or map Mp of the grossing specimen G can beprovided on a display. A user can select desired cut locations fortissue samples. The grossing system 100′ can generate a template ofdefined cut locations onto the specimen G based on the defined cut marks110 p on the planning image or map Mp. The system 100′ can be configuredto provide a light template that projects a tattoo type line or linesover the specimen G at defined cut locations. In some embodiments, thesystem 100′ can be configured to generate a color-based (printed orsprayed) tattoo which can be automatically applied with different dyecolors which may be defined by standard routines for different cutorientations at different cut locations. Image registration can be usedafter each cut is made or after all cuts are made to match an actual cutwith a virtual cut mark 110 based on the planning cuts 110 p as astarting point for the macroscopic image M. A set of releasablyattachable anchors 166 can be used to hold the specimen in position onthe board or tray, for example.

As discussed above with respect to FIG. 4, and as shown, for example inFIGS. 8-10, the viewer V can be configured to provide one or moremacroscopic images M that can include virtual cut location marks 110that can be viewed in connection with the slide images S.

The virtual cut location marks 110 can be provided as “cut referenceobjects” shown as overlays on the macroscopic image(s) M. These virtualcut location marks 110 can be electronically connected to thecorresponding slides S. Typically, the electronic connection can beshown in a visually distinct manner for easy intuitive visualrecognition by a user. Thus, for example, the viewer V canelectronically designate to a user which cut mark is associated with theslide under review.

In some embodiments, a set of slides S of a specimen G can be providedin one or more adjacent windows to a larger view of a tissue section ofa slide S under review, typically along a perimeter of the larger WSIview on the display.

Some or all of the slides S of a particular specimen G can be providedas thumbnail images in a GUI, which may include a touch screen GUI orother GUI input. The visual connections can be in the form ofhighlighting the virtual mark location 110 a (e.g., cut overlay)corresponding to the slide being viewed Sa and/or dimming the othervirtual cut marks 110.

Where there are thumbnail image overviews of the slides 200, acorresponding thumbnail T of the slide Sa can be visually changed in adefined manner relative to non-relevant slides to intuitively relay to auser which slide is under review and from which cut location the tissuesection on the slide was obtained. The visual change can be a visualenhancement (FIG. 8), e.g., visually enhanced to have a different coloror intensity or brightness relative to other slides in the thumbnailsand/or the other slides may be visually dimmed or faded (FIGS. 9, 10)and a respective associated cut mark location can be also visuallyenhanced (FIGS. 8, 9 and 10). The defined visual change can be via amoving border, use of an icon or arrow or at the thumbnail of the activeslide, a hovering mouse pointer or other defined visually distinctivechange feature.

FIGS. 8 and 10 illustrate an optional use of tether lines 88 thatconnect a cut location 110 on the macroscopic image M to a thumbnailslide image T and/or the larger displayed slide sample S. The visualtethers may be faded, dimmed or selected via a UI by a user. Typically,the tethers 88 are not required and may be used via a selection by auser or may be omitted as an option from the viewer V. Where used, thetethers may be just visual tethers or may be active links that allow formore information to be provided to a user for a particular cut locationand/or slide.

FIGS. 9 and 10 illustrate that a viewer V can show two large views ofslides S₁, S₂ in high magnification and the corresponding cut locationmarks 110 a (“A” and “E”) can be visually enhanced relative to the othercut mark locations. The cut location marks 110 a that are related to thecurrent large slide views S₁, S₂ can be shown as bold, with increasedintensity and/or in solid line and/or with a different color relative tothe other cut location marks 110 which can also be shown in brokenlines. Here, each relevant cut mark location 110 a is shown in abrighter, increased intensity and in a different solid line color on theimage M.

In some embodiments, as shown in FIG. 9, the two different relevant cutmark locations 110 a (here “A”, and “E”) may be shown in differentmanners, yet visually enhanced configurations from the non-relevant cutlocation marks 110 n, e.g., each may be shown with a different color 110v ₁, 110 v ₂, color-coded to the tissue slide thumbnail T₁, T₂ (Block A,Block E, respectively) and optionally with visual color indicia 110 v ₁,110 v ₂ provided on or adjacent the high magnification larger view ofS₁, S₂ for ease in intuitive visual association.

The virtual color location marks 115 (FIG. 5) can optionally be providedas digital “color reference objects” and may be shown as overlays on themacroscopic image(s). The virtual color marks 115 can be shown as GUImarkers in the image views.

Furthermore, the grossing sync system 100 and/or viewer V can beconfigured to automatically rotate a WSI image S such that a definedapplied (e.g., dye) color is presented in a defined orientation bydefault. This removes the need for manual orientation either at slidepreparation or at slide review, or, where orientation is notstandardized, can offer a more consistent slide visualization thatrelieves cognitive load from the pathologist. The viewer V (e.g., aviewer circuit or module) can be configured to rotate a scanned WSImicroscopic image according to dye color and present the WSI in a windowin a defined orientation, thereby providing a consistent slideorientation view based on location of a defined applied color in ascanned WSI.

FIG. 11 is a flow chart of exemplary steps that can be used to providethe integrated display of WSI slide images with a macroscopic image thatincludes cut location marks 110. As shown, grossing cut locations of aphysical specimen G can be digitally captured with respect to amacroscopic specimen image (block 300). Cut tissue is tracked duringlaboratory processing (block 305). Cut tissue is digitally connected toresulting microscopic slides (block 310). Concurrently at least one WSIimage and at least one macroscopic image are provided with cut locationson a display associated with a viewer (block 320). The viewer visuallyconnects the WSI image and a corresponding cut location in themacroscopic image (block 325).

FIG. 12 illustrates an exemplary workflow of making and using color,e.g., dye, information of a specimen G from a grossing station. Dyelocations can be digitally captured with respect to a macroscopicspecimen image (block 350). The cut tissue from the specimen is trackedduring laboratory processing (block 355). Cut tissue is digitallyconnected to resulting microscopic slides (block 360). Scanned WSIimages are digitally connected to cut tissue and a respectivemacroscopic image of the specimen (block 365). WSI image views areelectronically adjusted according to location of the dye in themacroscopic and/or microscopic images and the adjusted views areprovided to a display (block 370). Optionally, before or during theadjusting, dye locations are detected in respective (scanned) WSI images(block 372) and the adjusted WSI image and macroscopic image areintegrated into a viewer V for concurrent viewing in a display (e.g., ata workstation)(block 374).

FIGS. 13 and 14 are schematic illustrations of a viewer V incommunication with a viewer circuit Vc that can communicate with atleast one server 100S and at least one database 100D. The viewer circuitVc may be part of or communicate with the grossing sync system 100 whichcan include one or both of the cut mark location circuit 30 and the dyemark location circuit 31. The viewer Vc can have a client-serverconfiguration allowing various users to log into a portal hosted by aserver to operate the viewer V. The server 100S can communicate with anLIS system that may include some or all of the grossing specimentracking data or records 104 (FIG. 3), for example.

Referring to FIG. 13, the viewer V can comprise a UI 130 that allows auser to select various WSI presentations. As shown, viewing options caninclude dye color, stain color, cut location, rotate slide viewaccording to a selected or defined dye color, and a macroscopic imageoverlay with cut marks (on, off, fade, etc.).

As shown in FIGS. 13 and 14, the viewer circuit Vc can reside at leastin part on a server 100S that can be remote from a review/user site.Alternatively, the server 100S can be onsite or the viewer circuit Vccan partially or totally reside onboard a computer associated with aclinician workstation. The server 100S can be integrated into a singleserver or may be distributed into one or more servers or other circuitsor databases at a single physical site or at spatially separatelocations. Similarly, the viewer circuit Vc held by the one or moreservers 100S, and can be distributed into multiple processors ordatabases or integrated into one.

The viewer circuit Vc and/or server 100S may be embodied as a standaloneserver or may be contained as part of other computing infrastructures.The viewer circuit Vc and/or server 100S may be embodied as one or moreenterprise, application, personal, pervasive and/or embedded computersystems that may be standalone or interconnected by a public and/orprivate, real and/or virtual, wired and/or wireless network includingthe Internet, and may include various types of tangible, non-transitorycomputer-readable media. The viewer circuit Vc and/or server 100S mayalso communicate with the network via wired or wireless connections, andmay include various types of tangible, non-transitory computer-readablemedia.

The viewer circuit Vc and/or server 100S can be provided using cloudcomputing which includes the provision of computational resources ondemand via a computer network. The resources can be embodied as variousinfrastructure services (e.g., compute, storage, etc.) as well asapplications, databases, file services, email, etc. In the traditionalmodel of computing, both data and software are typically fully containedon the user's computer; in cloud computing, the user's computer maycontain little software or data (perhaps an operating system and/or webbrowser), and may serve as little more than a display terminal forprocesses occurring on a network of external computers. A cloudcomputing service (or an aggregation of multiple cloud resources) may begenerally referred to as the “Cloud”. Cloud storage may include a modelof networked computer data storage where data is stored on multiplevirtual servers, rather than being hosted on one or more dedicatedservers.

Users can communicate with the viewer circuit Vc and/or server 100S viaa computer network, such as one or more of local area networks (LAN),wide area networks (WAN) and can include a private intranet and/or thepublic Internet (also known as the World Wide Web or “the web” or “theInternet.” The viewer circuit Vc and/or server 100S can compriseappropriate firewalls (FIG. 14) for HIPPA or other regulatorycompliance.

FIG. 13 illustrates that viewer V can be in communication with (oronboard) at least one workstation W with a display or screen 35. Thedisplay 35 can include a UI 130 which can include touch, mouse, speech,cursor or other inputs that allows a user to select the viewing options.

The viewer circuit Vc or grossing sync system 100 can also include oneor more report output devices, including a display 35 (onboard theworkstation W or associated with another computer), a printer, afacsimile machine, and pervasive computer devices such as electronicnotepads, smartphones, cell phones and the like. A diagnosis based onthe analyzed sample can be delivered by email, facsimile, and/ordirectly to a HIS (Hospital Information System), LIMS (LaboratoryInformation Management System), PACS systems, or other systems ofaddresses (electronic or physical).

It is noted that while embodiments of the present invention use a remoteserver for the image enhancement, it is contemplated that differentclinic sites or each facility or room may have a dedicated on siteviewer with an image enhancement analysis circuit.

Embodiments of the present invention may take the form of an entirelysoftware embodiment or an embodiment combining software and hardwareaspects, all generally referred to herein as a “circuit” or “module.”Furthermore, the present invention may take the form of a computerprogram product on a (non-transient) computer-usable storage mediumhaving computer-usable program code embodied in the medium. Any suitablecomputer readable medium may be utilized including hard disks, CD-ROMs,optical storage devices, a transmission media such as those supportingthe Internet or an intranet, or magnetic storage devices. Some circuits,modules or routines may be written in assembly language or evenmicro-code to enhance performance and/or memory usage. It will befurther appreciated that the functionality of any or all of the programmodules may also be implemented using discrete hardware components, oneor more application specific integrated circuits (ASICs), or aprogrammed digital signal processor or microcontroller. Embodiments ofthe present invention are not limited to a particular programminglanguage.

Computer program code for carrying out operations of data processingsystems, method steps or actions, modules or circuits (or portionsthereof) discussed herein may be written in a high-level programminglanguage, such as Python, Java, AJAX (Asynchronous JavaScript), C,and/or C++, for development convenience. In addition, computer programcode for carrying out operations of exemplary embodiments may also bewritten in other programming languages, such as, but not limited to,interpreted languages. Some modules or routines may be written inassembly language or even micro-code to enhance performance and/ormemory usage. However, embodiments are not limited to a particularprogramming language. As noted above, the functionality of any or all ofthe program modules may also be implemented using discrete hardwarecomponents, one or more application specific integrated circuits(ASICs), or a programmed digital signal processor or microcontroller.The program code may execute entirely on one (e.g., a workstationcomputer), partly on one computer, as a standalone software package,partly on the workstation's computer or Scanner's computer and partly onanother computer, local and/or remote or entirely on the other local orremote computer. In the latter scenario, the other local or remotecomputer may be connected to the user's computer through a local areanetwork (LAN) or a wide area network (WAN), or the connection may bemade to an external computer (for example, through the Internet using anInternet Service Provider).

The present invention is described in part with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing some or all of thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowcharts and block diagrams of certain of the figures hereinillustrate exemplary architecture, functionality, and operation ofpossible implementations of embodiments of the present invention. Inthis regard, each block in the flow charts or block diagrams representsa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that in some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the figures. For example, two blocks shown in successionmay in fact be executed substantially concurrently or the blocks maysometimes be executed in the reverse order or two or more blocks may becombined, depending upon the functionality involved.

As illustrated in FIG. 15, embodiments of the invention may beconfigured as a data processing system 116 that is in communication withor forms part of the viewer circuit Vc. The data processing system caninclude at least one processor 400, memory 436 and input/output circuits446. The data processing system may be incorporated in, for example, oneor more of a personal computer, database, workstation W, server, routeror the like. The system 416 can reside on one machine or be distributedover a plurality of machines. The processor 400 communicates with thememory 436 via an address/data bus 448 and communicates with theinput/output circuits 446 via an address/data bus 449. The input/outputcircuits 446 can be used to transfer information between the memory(memory and/or storage media) 436 and another computer system or anetwork using, for example, an Internet protocol (IP) connection. Thesecomponents may be conventional components such as those used in manyconventional data processing systems, which may be configured to operateas described herein.

In particular, the processor 400 can be commercially available or custommicroprocessor, microcontroller, digital signal processor or the like.The memory 436 may include any memory devices and/or storage mediacontaining the software and data used to implement the functionalitycircuits or modules used in accordance with embodiments of the presentinvention. The memory 436 can include, but is not limited to, thefollowing types of devices: ROM, PROM, EPROM, EEPROM, flash memory,SRAM, DRAM and magnetic disk. In some embodiments of the presentinvention, the memory 436 may be a content addressable memory (CAM).

As further illustrated in FIG. 15, the memory (and/or storage media) 436may include several categories of software and data used in the dataprocessing system: an operating system 452; application programs 454;input/output device drivers 458; and data 456. As will be appreciated bythose of skill in the art, the operating system 452 may be any operatingsystem suitable for use with a data processing system, such as IBM®,OS/2®, AIX® or zOS® operating systems or Microsoft® Windows®95,Windows98, Windows2000 or WindowsXP operating systems, Unix or Linux™,IBM, OS/2, AIX and zOS are trademarks of International Business MachinesCorporation in the United States, other countries, or both while Linuxis a trademark of Linus Torvalds in the United States, other countries,or both. Microsoft and Windows are trademarks of Microsoft Corporationin the United States, other countries, or both. The input/output devicedrivers 458 typically include software routines accessed through theoperating system 452 by the application programs 454 to communicate withdevices such as the input/output circuits 446 and certain memory 436components. The application programs 454 are illustrative of theprograms that implement the various features of the circuits and modulesaccording to some embodiments of the present invention. Finally, thedata 456 represents the static and dynamic data used by the applicationprograms 454 the operating system 452 the input/output device drivers458 and other software programs that may reside in the memory 436.

The data 456 may include (archived or stored) digital WSI and/ormacroscopic image data sets 426 correlated to respective patients. Asfurther illustrated in FIG. 15, according to some embodiments of thepresent invention, the application programs 454 include a grossing imagesynch Module 424 that can provide virtual cut location marks associatedwith tissue taken from cut locations on a grossing specimen G. Theapplication programs can include a cut location marking Module 425. Theapplication programs can optionally include a dye correlation Module 426for correlating dye color to locations on a specimen. The applicationprograms can optionally include an image registration module 428 thatcan adjust cut mark locations due to specimen shape change. Theapplication programs can optionally include a dye color slideorientation Module 429 for a viewer. The data interface module can bedecoupled or isolated from the visualization/viewer module. Theapplication program 454 may be located in a local server (or processor)and/or database or a remote server (or processor) and/or database, orcombinations of local and remote databases and/or servers.

While the present invention is illustrated with reference to theapplication programs 454, and Modules 424, 425, 426, 428 and 429 in FIG.15, as will be appreciated by those of skill in the art, otherconfigurations fall within the scope of the present invention. Forexample, rather than being application programs 454 these circuits andmodules may also be incorporated into the operating system 152 or othersuch logical division of the data processing system. Furthermore, whilethe application programs 424, 425, 426, 428 and 429 are illustrated in asingle data processing system, as will be appreciated by those of skillin the art, such functionality may be distributed across one or moredata processing systems in, for example, the type of client/serverarrangement described above. Thus, the present invention should not beconstrued as limited to the configurations illustrated in FIG. 15 butmay be provided by other arrangements and/or divisions of functionsbetween data processing systems. For example, although FIG. 15 isillustrated as having various circuits and modules, one or more of thesecircuits or modules may be combined or separated without departing fromthe scope of the present invention.

The foregoing is illustrative of the present invention and is not to beconstrued as limiting thereof. Although a few exemplary embodiments ofthis invention have been described, those skilled in the art willreadily appreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention as defined inthe claims. The invention is defined by the following claims, withequivalents of the claims to be included therein.

That which is claimed:
 1. A histopathology and/or cytopathology viewer,comprising: a display; and a viewer circuit configured to cause thedisplay to concurrently display: (i) a macroscopic view of a grossingspecimen with virtual cut location marks associated with orientation andlocation of actual physical cut locations used to obtain tissue samplesfrom the grossing specimen and (ii) at least one digital microscopicwhole-slide image (WSI) of a tissue section from the specimen, wherein,for the at least one WSI image on the display, a relevant cut locationmark is shown visually enhanced from other cut location marks on themacroscopic view to thereby allow a user to visually connect where thetissue section in the WSI image was obtained during a grossingprocedure.
 2. The viewer of claim 1, wherein the viewer circuit isconfigured to visually change a display parameter of at least one of:(a) a virtual cut location mark on the macroscopic view that isassociated with a location on the specimen from which a tissue sectionof the at least one WSI in the viewer display was obtained; or (b)virtual cut location marks on the macroscopic view that are notassociated with the tissue section of the at least one WSI in the viewerto thereby visually enhance the relevant cut mark location.
 3. Theviewer of claim 1, further comprising providing thumbnail WSI images ofslides associated with the grossing specimen of a patient and visuallychanging a display parameter of a thumbnail associated with the at leastone WSI in the viewer.
 4. The viewer of claim 1, wherein a perimeter ofthe thumbnail or thumbnails associated with the at least one WSI in theviewer display is visually enhanced and/or other thumbnails are dimmedor faded in the display window, wherein the virtual cut location marksare provided as overlay linear objects on the macroscopic view, andwherein a virtual cut location mark on the macroscopic view that isassociated with a location on the specimen from which a tissue sectionof the at least one WSI in the viewer display was obtained is visuallyhighlighted, increased in intensity or changed in color relative toother virtual cut mark locations to thereby visually connect the WSIimage in the view to a correct virtual cut location mark.
 5. The viewerof claim 1, wherein the viewer circuit is configured to analyze scannedWSI images for determining color and color location and adjust anorientation of the WSI image for the viewer to consistently provideviews of the WSI images to the display in a common orientation withrespect to a tissue sample location and orientation in or on thegrossing specimen based on the determined color.
 6. A grossingworkstation, comprising: a cut mark location identification circuit; atleast one camera over a workspace at the workstation in communicationwith the cut mark location identification circuit; and a cuttinginstrument in communication with the cut mark location identificationcircuit; wherein the cut mark location identification circuit isconfigured to identify physical cut mark locations on a grossingspecimen and generate corresponding virtual cut location marks on amacroscopic image or model of the specimen for a viewer, and wherein arespective virtual cut location mark corresponds to a physical cutlocation made by the cutting instrument on the grossing specimen used toobtain a tissue sample such that the virtual cut location marks aregiven a correct spatial position on the macroscopic image or model ofthe specimen for the viewer.
 7. The workstation of claim 6, wherein thecut mark location circuit analyzes a plurality of macroscopic images ofthe specimen obtained by the at least one camera during a grossingprocedure of the specimen, including at least one base macroscopic imageobtained prior to a physical cutting to obtain a tissue sample, andwherein the cut mark location identification circuit is configured toelectronically place the virtual cut location marks as overlay cutlocation objects on the macroscopic image or model of the specimen forthe viewer.
 8. The workstation of claim 6, wherein the cut mark locationidentification circuit is configured to electronically track movement ofthe cutting instrument in the workspace.
 9. The workstation of claim 6,wherein the cut mark location identification circuit is configured toemploy image recognition of movement and location of the cuttinginstrument in images obtained by the at least one camera to generate thevirtual cut mark locations on the macroscopic image or model.
 10. Theworkstation of claim 6, wherein the cutting instrument is configuredwith a defined leading end portion shape which has a distinctconspicuous visual appearance detectable by image recognition to allowthe cut mark location identification circuit to identify movement andcontact of the cutting instrument for identifying the placement andorientation of the respective virtual cut location marks.
 11. Theworkstation of claim 6, wherein the cut mark location circuit isconfigured to employ a hands-free command or input from a user toinitiate identifying a cut location.
 12. A method of obtaining and/orprocessing digital pathology and cytology images for viewing,comprising: electronically identifying when a cutting instrument isapproaching and/or on a grossing specimen; and automaticallyelectronically generating virtual cut location marks on a digitalmacroscopic image or model of a grossing specimen based on theelectronic identification, wherein a respective virtual cut locationmark corresponds to a physical cut location made by the cuttinginstrument on the grossing specimen used to obtain a tissue sample suchthat the virtual cut location is given a correct spatial position on themacroscopic image or model of the specimen.
 13. The method of claim 12,further comprising electronically automatically obtaining a plurality ofmacroscopic images of the specimen during a grossing procedure of thespecimen, including at least one base macroscopic image obtained priorto one or more physical cutting to obtain a tissue sample, and placingthe virtual cut location marks on one or more of the obtainedmacroscopic images or model of the specimen for display in a viewer. 14.The method of claim 12, wherein the electronic identification is carriedout by electronically tracking movement of the cutting instrument. 15.The method of claim 13, wherein the electronic identification is carriedout using image recognition of movement and location of the cuttinginstrument in the obtained images.
 16. The method of claim 14, whereinthe electronic identification of the cutting instrument is carried outusing image recognition of a defined leading end portion configurationwhich is configured as to have a defined distinct conspicuous visualappearance detectable by image recognition.
 17. The method of claim 12,wherein the electronic identification is initiated with a hands-freecommand to a cut location identification circuit in communication with agrossing workstation.
 18. The method of claim 12, wherein the electronicidentification is initiated by the cutting instrument transmitting awireless signal at a time of a cut.
 19. The method of claim 12, whereinthe automatic electronic generation of the virtual cut location markscomprises electronically generating an electronic overlay of at leastone cut location object provided as a linear mark and displaying theoverlay on a macroscopic digital image or model of the specimen.
 20. Themethod of claim 12, further comprising concurrently displaying (i) amacroscopic image of the specimen with the virtual cut location markswith (ii) at least one digital microscopic whole-slide image (WSI) oftissue from the specimen, wherein a virtual cut location mark associatedwith the at least one digital WSI is visually enhanced relative to othercut location marks when it is associated with where a tissue sectionfrom the WSI was obtained from the specimen.
 21. The method of claim 20,wherein the virtual cut location marks are shown as respective overlayson the macroscopic image with the visual enhancement provided byhighlighting a virtual cut location mark whenever it corresponds to amicroscopic WSI image in a viewer.
 22. The method of claim 12, furthercomprising: electronically automatically obtaining a plurality ofmacroscopic images of the specimen during a grossing procedure of thespecimen, including at least one base macroscopic image obtained priorto any physical cutting to obtain tissue samples, and placing thevirtual cut location marks on one or more of the obtained macroscopicimages or a model of the specimen for a viewer macroscopic image; andelectronically adjusting for movement of the specimen during a grossingprocedure using the base image and one or more of the subsequentplurality of images to register respective virtual cut locations to theviewer macroscopic image or model.